Image capture with digital light path length modulation

ABSTRACT

An image capture system comprising an optical path length extender (OPLE) to direct the light having a first polarization through a first light path through the OPLE, and to direct the light having a second polarization through a second light path through the OPLE, the first and second light paths having different light path lengths. The image capture system further comprising an image sensor to capture a plurality of image portions at a plurality of focal distances.

RELATED APPLICATIONS

The present invention claims priority to U.S. patent application Ser.No. 15/236,101, filed on Aug. 12, 2016. The present invention alsoclaims priority to U.S. patent application Ser. No. 15/358,040 filed onNov. 21, 2016. The above applications are incorporated herein byreference.

FIELD

The present invention relates to an image capture system, and moreparticularly to an image capture system utilizing light path lengthmodulation.

BACKGROUND

Providing multiple focal planes, or discrete steps of focus adjustment,is useful for a number of applications. It can be part of capturingthree dimensional data. In the prior art, multiple focus captureutilized mechanical movement such as gears or liquid lenses. Suchmechanisms are expensive, slow, and relatively fragile. Another priorart method of capturing multiple focal lengths involves using multiplecameras. However, this is bulky and expensive. The bulk and expense alsolimits the number of focal lengths that can be simultaneously captured.

BRIEF DESCRIPTION OF THE FIGURES

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is an overview diagram of a camera system in which the presentinvention may be used.

FIG. 2 is a block diagram of one embodiment of an image capture system.

FIG. 3A is a block diagram of one embodiment of a capture assembly.

FIG. 3B is a diagram of one embodiment of an exemplary modulation stackincluding a plurality of digital light path length modulators.

FIG. 4A is a diagram of one embodiment of using a camera system forcapturing light with a first state, without reflection

FIG. 4B is a diagram of one embodiment of using the camera system ofFIG. 4A for light capturing light with a second state with reflection.

FIG. 5A and 5B are two embodiments of OPLEs which may be used in adigital light path length modulator.

FIG. 5C is one embodiment of an OPLE striped for selective capture.

FIG. 5D illustrates one embodiment of the light paths in a striped OPLE.

FIG. 5E is a diagram of one embodiment of an image capture system usinga striped OPLE.

FIG. 5F is a diagram of one embodiment of an image display system usinga striped OPLE.

FIG. 6 is a flowchart of one embodiment of an overview of image captureusing the modulation stack.

FIG. 7 is a more detailed flowchart of one embodiment of capturing animage using the system.

FIG. 8 is a flowchart of one embodiment of using a striped OPLE tocapture images.

FIG. 9 is a flowchart of one embodiment of applications of the imagecapture system disclosed.

FIG. 10 is a block diagram of one embodiment of a computer system thatmay include a memory and processor to enable post-processing.

DETAILED DESCRIPTION

An image capture system is described. The image capture system in oneembodiment includes a modulation stack, with one or more digital lightpath length modulators. Each digital light path length modulatorincludes an optical path length extender (OPLE) and a polarizationmodulator, and can be used to adjust the path length of light. In oneembodiment, light with state 1 polarization travels through a longerpath in the OPLE than light with state 2 polarization. This can be usedto capture data at two or more focal distances. An OPLE is made up ofone or more plates with a plurality of polarization sensitive reflectiveelements. A plurality of digital light path length modulators create themodulation stack. In one embodiment, the image capture system mayinclude a striped OPLE to capture images at two focal distances.

This mechanism can be used for cameras and image capture, and variousother uses in which light waves or other waves in a similar spectrum arecaptured. Cameras may include conventional cameras with both fixed andinterchangeable lenses, including but not limited to cell phone camerasand DSLR cameras, medical cameras such as endoscopes, and other types ofprobes or cameras which acquire image data.

The following detailed description of embodiments of the invention makesreference to the accompanying drawings in which like references indicatesimilar elements, showing by way of illustration specific embodiments ofpracticing the invention. Description of these embodiments is insufficient detail to enable those skilled in the art to practice theinvention. One skilled in the art understands that other embodiments maybe utilized and that logical, mechanical, electrical, functional andother changes may be made without departing from the scope of thepresent invention. The following detailed description is, therefore, notto be taken in a limiting sense, and the scope of the present inventionis defined only by the appended claims.

FIG. 1 is an overview diagram of a camera system in which the presentinvention may be used. The exemplary camera 100 is a standard digitalcamera, in one embodiment. In one embodiment, the image capture systemin a camera includes a detachable lens. The focus distance adjustmentmay be embodied in various parts of the camera. In FIG. 1, it is shownin four different positions 110, 120, 130, and 140. The focus distanceadjustment may be integral to the lens, as shown with element 110. Thefocus distance adjustment may be external to lens element but part ofthe lens, as shown in element 120. The focus distance adjustment may beintegral to the camera, as shown in element 130. The focus distanceadjustment may also be in front of the sensor, a shown in element 140.In one embodiment, the focus distance adjustment positioned in front ofthe sensor may be a striped OPLE, as will be described in more detailbelow. In one embodiment, there may be multiple focus distanceadjustments, in a single camera/image capture system.

In one embodiment, by positioning the focus distance adjustments withinthe camera lens assembly, the system can be used to retrofit an existingcamera system for capturing multiple focal points utilizing a single CCD(charged coupled device) or other image capture mechanism. The camerasystem 100 is merely an example of an image capture system.

FIG. 2 is a block diagram of one embodiment of an image capture system.The image capture system 200 includes, in one embodiment, a processor270, memory 275, and I/O system 280, as well as an image capturesubsystem 205, and optionally an auxiliary control system 285. In oneembodiment, the system also includes a display system 299, which enablesthe system 200 to display the captured data.

Processor 270 in one embodiment may be used to control the image capturesub-system 205, to adjust the light path lengths. The processor 270 mayalso be used to post-process the captured data, in one embodiment.Memory 275 may be a buffer-memory, enabling the image capture system 200to capture image and video data. I/O system 280 makes the captured imagedata available to other systems. In one embodiment, post-processingtakes place in a separate device, such as a computer or server system,which provides post-processing for the captured image data.

The image capture subsystem 205 includes, in one embodiment, captureassembly 220, modulation stack 230, and imager system 255. The lens 215in one embodiment is a conventional lens to capture an image.

The polarizer 225 allows light with a particular polarization to passthrough. In one embodiment, the polarization may be S/P polarization, orcircular polarization. The polarizer 225 in one embodiment is a linearpolarizer. The polarizer 225 in one embodiment is a linear polarizerwith a backwards circular ¼ wave polarizer. Such a combined polarizeraddresses glare issues.

The capture assembly 220 includes a modulation stack 230. The modulationstack 230 includes one or more digital light path length modulators 235,240. The digital light path length modulators 235, 240 alter the lightpath length based on the polarization of the light. In one embodiment,polarizer 225 may be positioned after modulation stack 230.

Capture assembly 220 is used to capture images at multiple focaldistances. In one embodiment, the image capture subsystem 205 mayinclude additional mechanical and optical elements 250 which can providecorrection or alteration of the image.

The image data is captured by imager system 255. Imager system 255includes an image sensor 260. In one embodiment, it further includes adigital correction system 265. In another embodiment, the digitalcorrection may take place in a different system from the image capturesystem 200. This may include, in one embodiment, reconstructing theimage(s), based on the captured data, and data from the modulationstack.

In one embodiment, the system may receive data from auxiliary datasystem 285, which may be used to control the image capture sub-system205, directly or through processor 270. The auxiliary data system 285may provide information for selecting the appropriate focal lengths. Asnoted above, the modulation stack 230 allows image elements to becaptured at various focal distances. The auxiliary control system 285may be used to select the focal distances, based on various factors.

One auxiliary control system may be a content-based focal distanceselection 290, which enables the system to selectively choose one ormore focal distances, based on what is being captured. For example, thesystem may selectively choose a portion of the image data for focus. Inone embodiment, content-based focal distance selection 290 may utilize adistance sensor, to detect a distance, for capture. One example of adistance sensor that may be used is a time of flight sensor. Based onthe type of system, the focal distances selection may vary. For example,to capture a fingerprint including blood flow data, the system may sensea distance to the finger, and then identify a second focal distancebeyond that distance. In one embodiment, the user input system 292 mayenable a user to select the use of the image capture system 200. In oneembodiment, such selection may be made via a remote system, such as amobile device paired with the image capture system 200.

User input systems 292 enable focus selection based on user input,through various mechanisms. Such user input may be provided via gesture,voice control, an adjustment dial or another selection mechanism. Asnoted above, the user input system 292 may be coupled with thecontent-based focal distance selection 290, such that the user entersthe targeted image capture and purpose, and the actual focal distancesare selected automatically. User input systems 292 may include videogame controllers, microphones, cameras, inertial measurement sensors,coupled devices, and other sensors for detecting user input.

Other control data 293 may also be provided to the system. Any of thisdata from auxiliary control system 285 may be used to adjust the focaldistances for the one or more captured image elements. In oneembodiment, in addition to auxiliary data system 285, the system mayadditionally accept manual adjustment 295.

In one embodiment, the image capture system 200 may capture multiplefocal distances within a single image, utilizing the polarizer and astriped OPLE, as will be described below.

In one embodiment, the image capture system 200 may include a displaysystem 299 to display the captured image. In one embodiment, the displaysystem 299 may be a conventional display of a camera, showing what imageis being captured. In one embodiment, the display system 299 may utilizea near-eye display system or other system which enables display ofmultiple image elements at multiple virtual distances.

FIG. 3A is a block diagram of one embodiment of a capture assembly 310.The capture assembly 310, in one embodiment, includes a plurality ofdigital light path length modulators (335, 345) as well as a pluralityof intermediate optics elements (330, 340, 350, 355) together forming agreater modulation stack 315. In one embodiment, the capture assembly310 in a real system may include 6-30 elements which include lenses,mirrors, apertures, and the like, referred to as intermediate optics. Inone embodiment, the intermediate optics may be interspersed with thedigital light path length modulators. In one embodiment, they may bepositioned before and/or after the set of digital light path lengthmodulators. In one embodiment, polarization filter 320, 325 may bepositioned before (320) or after (325) the greater modulation stack 315.

In one embodiment, the capture assembly 310 may correct for chromaticaberration and other irregularities of optical systems.

FIG. 3B illustrates one embodiment of a modulation stack 370 includingthree digital light path length modulators. Each of the digital lightpath length modulators 372, 382, 392 includes a polarization modulatorand an OPLE. In this example, one of the OPLEs 384 is a self-alignedOPLE. In this example, the first OPLE 374 is a longitudinal OPLE, whilethe other OPLEs are transverse OPLEs. One of the transverse OPLEs 384 isa self-aligning OPLE. These OPLEs are described in co-pending U.S.patent application Ser. No. 15/236,101, filed on Aug. 12, 2016 and U.S.patent application Ser. No. 15/358,040 filed on Nov. 21, 2016,incorporated herein by reference.

With the shown set of three different OPLEs, the system can create up toeight (2³) virtual object distances by selectively modulating thepolarization, as follows:

OPLE 1 OPLE 2 OPLE 3 State 1 State 1 State 1 State 1 State 1 State 2State 1 State 2 State 1 State 1 State 2 State 2 State 2 State 1 State 1State 2 State 1 State 2 State 2 State 2 State 1 State 2 State 2 State 2

In one embodiment, the configuration utilizes two OPLEs to be able tocapture image elements at four focal distances.

FIGS. 4A and 4B are diagrams of one embodiment of a digital light pathlength modulator in a camera system. The camera lens 410 captures theimage data, and transmits it through the polarizing filter 420. Thepolarization modulator 430 selectively polarizes the light, and sends itthrough OPLE 440. The output from OPLE 440 goes to image sensor 450. Thepolarization modulator 430 is “off” in FIG. 4A, and the state 2polarized light is not modulated. The OPLE 440 does not reflect state 2polarized light, and thus the light passes straight through the digitallight path length modulator 460. The light coming out digital light pathlength modulator 460 impacts the image sensor 450. Of course, though itis not shown, additional optical elements may be included in thissystem, including lenses, correction systems, etc.

FIG. 4B shows the same system when the polarization modulator 430 is on,and modulates the light to state 1 polarization. The state 1 polarizedlight goes through a longer light path, because it is reflected by thepolarization sensitive reflective elements in the OPLE 440. This willcause objects at a nearer distance to come into focus without moving anyelements of the imaging lens. Also, focus can be changed as fast as thepolarization modulator can change states, which commonly can be under 50milliseconds. The OPLE 440 and polarization modulator 430 form a digitallight path length modulator 460. In one embodiment, the system mayinclude a plurality of digital light path length modulators 460.

FIG. 5A and 5B are two embodiments of OPLEs which may be used in adigital light path length modulator. The two OPLEs are referred to astransverse OPLE (OPLE Type #1, FIG. 5A) and longitudinal OPLE (OPLE Type#2, FIG. 5B).

FIG. 5A is a diagram of one embodiment of a first type of OPLE, referredto as a transverse OPLE. The OPLE includes one or more plates, eachplate having a plurality of polarization sensitive reflective elements,which reflect light having a first polarization, and pass through lightwith a second polarization. The reflected light bounces between thepolarization sensitive reflective elements two or more times, beforeexiting the OPLE. This increases the path length of the light having thefirst polarization, compared to the light having the second polarizationwhich passes directly through the transverse OPLE. Further details onthe OPLE of FIG. 5A are discussed in co-pending U.S. patent applicationSer. No. 15/236,101, filed on Aug. 12, 2016, which is incorporatedherein in its entirety.

FIG. 5B is a diagram of one embodiment of a second type of OPLE,referred to as a longitudinal OPLE. The OPLE includes a reflectiveelement on the bottom surface, which reflects light having a firstpolarization. The light in turn bounces back from the top of the OPLE,before exiting the OPLE through the bottom surface. This increases thepath length of the light having the first polarization, compared to thelight having the second polarization which passes directly through thelongitudinal OPLE. Further details on the OPLE of FIG. 5B are discussedin co-pending U.S. patent application Ser. No. 15/358,040 filed on Nov.21, 2016, which is incorporated herein in its entirety.

FIG. 5C is one embodiment of an OPLE striped for selective capture. TheOPLE 530 has a plurality of stripes 540. In one embodiment, the stripesmatch the spacing between the reflective elements (d). In oneembodiment, the stripes may extend slightly beyond the edge of thereflective element.

The stripes 540 provide a light blocking coating on the entry surface ofthe OPLE. In one embodiment, the light blocking coating is a metal. Inone embodiment, the light blocking coating is aluminum. The polarizationcoating on the OPLE enables the striping of images at two focaldistances onto a single image capture apparatus. The separate images canthen be reconstructed in post-processing.

In one embodiment, the striped OPLE may assist in capturing multiplefocal distances simultaneously. In one embodiment, additionally oralternatively, focal distances may be captured time sequentially, e.g.with different focal distances captured as the length of the image pathis changed.

In one embodiment, for simultaneous capture of images, thereconstruction may utilize edge finding to identify the separationbetween the focal planes, which have a magnification difference.

FIG. 5D illustrates one embodiment of the light paths using a stripedOPLE. As can be seen, the striped OPLE 550 receives both State 1 andState 2 incoming light. When the incoming light hits a portion of theOPLE 550 covered by a light blocking stripe 560, it does not passthrough the OPLE 550. When the incoming light hits a portion of the OPLE550 not covered with the light blocking stripe, it passes through theOPLE 550. As discussed above, state 1 light is reflected by thepolarization sensitive reflective elements, while state 2 light passesthrough. Thus, state 1 light impacts the image sensor 570 in differentlocations than state 2 light, and the image captured by the image sensor570 has separate stripes of image data, for state 1 and state 2elements. These stripes may be separated to construct two images or amultifocal image.

FIG. 5E is a diagram of one embodiment of an image capture system usinga striped OPLE. The image capture system 580 receives incoming light582, which is passed to a striped OPLE 583. The striped OPLE 583 stripesthe image elements onto image sensor 584. In one embodiment, the OPLE583 is positioned in close proximity to image sensor 584. In oneembodiment, the distance between OPLE 583 and image sensor 584 is 0.1mm. In one embodiment, the distance may range from 0 mm to 1 cm. In oneembodiment, the striped OPLE may be coupled to, or integrated into, theimage capture device.

As shown in FIG. 5D, the P-polarized image portions are separated fromthe S-polarized image portions. The image sensor 584 receives the image.Digital correction system 585 creates the image(s) from the receiveddata. The image may be a multi-focal image, or two images at differentfocal distances. The different focal distances are controlled by thedifference in light path lengths for the striped OPLE 583. In oneembodiment, the image correction may be done using processor 586. In oneembodiment, memory 587 may store the data, and I/O system 588 maycommunicate the data outside the image capture system 580. In oneembodiment, additional optical elements, and optionally additional OPLEsor modulation stack elements may be incorporated into the system.

FIG. 5F is a diagram of one embodiment of an image display system usinga striped OPLE. The striped OPLE configuration may also be utilized in adisplay system. In a display system the light comes from light source592. In one embodiment, the light source may be a spatial lightmodulator 592. In one embodiment, the light output by the light source592 is controlled by digital correction system 591.

Striped OPLE 593 is positioned in proximity to the imaging assembly 594.The striped OPLE separates the portion of the light with S-polarizationfrom the portion of the light with P-polarization, and has a longerlight path for one of those polarizations. The imaging assembly 594utilizes the image data to display image elements at two focaldistances, corresponding to the two polarizations.

In one embodiment, the image correction and image assembly may be doneusing processor 596. In one embodiment, memory 597 may store the data,and I/O system 598 may obtain the data to be displayed from outside thedisplay system 580. In one embodiment, additional optical elements, andoptionally additional OPLEs or modulation stack elements may beincorporated into the system.

FIG. 6 is a flowchart of one embodiment of an overview of image captureusing the modulation stack. The process starts at block 610. In oneembodiment, the process starts when the system is turned on to captureimages. In one embodiment, while the flowchart shows a single imagecaptured at multiple focal distances, the system may be used to capturemultiple images in a series, such as video capture. In one embodiment,the system can capture video images at multiple focal distances in oneembodiment.

At block 620 the system selects one or more focal planes for capture. Inone embodiment, the selection may be made manually by a user. In oneembodiment, the selection may be made automatically by the system basedon an evaluation of the image, as discussed above. In one embodiment,the focal planes may be a plurality of discrete focal steps, coveringthe real-world image being captured.

At block 630, the first image/subframe is captured with a firstpolarization, at first focal distance. At block 640, the secondimage/subframe is captured with a second polarization at a second focaldistance. In one embodiment, the first and second images may be capturedsimultaneously. In another embodiment, the first and second images maybe captured sequentially.

At block 650, the images are reconstructed. In one embodiment, a seriesof images captured at a plurality of focal lengths may be combined tocreate a single multi-dimensional image. In one embodiment,simultaneously captured images may be reconstructed by separating theimage portions at the plurality of focal distances. In one embodiment,other adjustments may be made to the image, including color correction,optical correction, etc. This may be done by a processor on the imagecapture device, or on a separate computer system.

At block 660, the captured image data is stored and/or displayed. In oneembodiment, the image data may be stored in memory. In one embodiment,interim image data may be stored in a cache memory, and the final imagedata may be stored in memory, which may be a flash memory, a hard drive,or other type of storage. In one embodiment, the displaying may be usinga near-eye display system or other system capable of displayingmulti-focal images. The process then ends at block 670. In oneembodiment, if the system is capturing video or a stream of images, thesystem may automatically return to block 620, to select subsequent focalplanes, and capture additional images, until it is turned off orotherwise terminated. Although this flowchart only shows twoimage/subframes being captured, it should be understood that any numberof images/subframes may be captured, at different focal distancesutilizing the described system and process.

FIG. 7 is a more detailed flowchart of one embodiment of capturing animage using the system. The process starts at block 710.

At block 720, the system selects one or more focal planes for imagecapture. In one embodiment, for a system providing simultaneous capture,a plurality of simultaneously captured focal planes may be selected. Inone embodiment, for a system providing sequential capture, the firstfocal plane for capture may be selected.

At block 730, the system adjusts the modulation stack for the selectedfocal planes. As discussed above, two potential focal planes may beselected for simultaneous capture, utilizing a striped OPLE, orutilizing a polarizer.

At block 740 the image(s) are captured.

At block 750, the process determines whether all of the images to becaptured have been captured. If not, the process returns to block 730 toadjust the light path length through the modulation stack. As notedabove this is done by altering the polarization of the light. If allimages have been captured, the process continues to block 760.

At block 760, the process determines whether there are anysimultaneously captured images. If not, at block 780, the captured imagedata is stored and/or displayed, and the process ends at block 790.

If there were simultaneously captured images, the process continues toblock 770. The simultaneous images are captured on the same CCD/sensor,so the system, at block 770 reconstructs the individual images insoftware. In one embodiment, this may be done by identifying the edgesof each captured image segment, separating the image segments by focaldistance, and combining the image segments to generate an image. Theprocess then continues to block 780.

FIG. 8 is a flowchart of one embodiment of using a striped OPLE tocapture images. The process starts at block 810.

At block 820, image data is received through a striped OPLE. In oneembodiment, this configuration includes only a single OPLE, which isstriped. In one embodiment, the striping is as shown in FIG. 5C. Inanother embodiment, this configuration may include a plurality of OPLEs.In one embodiment, only one of the plurality of OPLEs is striped. In oneembodiment, the final OPLE is striped.

At block 830, the image is captured. As noted above, the image capturestwo focal distances in a single image, separated by polarization, basedon the striping.

At block 840, the captured images are reconstructed. The reconstructionof the images in one embodiment creates two separate complete images,having different focal distances.

At block 850, the captured images are stored and/or displayed. Theprocess then ends at block 860.

FIG. 9 is a flowchart of one embodiment of applications of the imagecapture system disclosed. The image capture system described can capturevarious types of images, which may be used for the creation ofadditional data about the scene which is captured. As noted above, theimage being captured can range from a nature scene, to a medical images(for example, capturing a surgical site at multiple distances),microscopy, biometric images (for example capturing a fingerprint imageand an image of the blood flow underneath the fingerprint), to aplurality of individuals standing at different distances, etc.

At block 920, the process determines whether the image capture system iscapable of processing the data locally. There may be image capturesystems that have integrated processors capable of processing the datalocally. However, some systems cannot do so. For example, the system maybe implemented as a retrofit lens for a camera. In such an embodiment,the modulation stack may be built into a camera lens assembly, enablingretrofitting of existing camera bodies for multi-focal capture. Suchsystems cannot perform post-processing. There are other embodiments inwhich the camera cannot perform processing, or can only do some of thepotential processing needed to generate the final images.

If the camera can process locally, in one embodiment, it does so, andthe process continues directly to block 930. If the image capture devicecannot complete processing locally, at block 925, the post-processing,such as image separation, and building of multi-dimensional images, isimplemented in a separate system. In one embodiment, the processing maybe split between the computer and the image capture device, with theimage capture device doing pre-processing.

In one embodiment, the separate system may be a client computer. In oneembodiment, the separate system may be a server computer. If a servercomputer is used, in one embodiment, the user (or user's computer orother device) may upload the image data to a server, and receive theprocessed image data from the server. The process then continues toblock 930.

At block 930, the process determines whether image capture wassimultaneous. When image capture is simultaneous a single sensorcaptures interleaved slices of images at more than one focal distance.In such an instance the images are separated in post-processing, atblock 935. The process then continues to block 940.

At block 940, the process determines whether the capture isdifferentiated by wavelength. In one embodiment, the images may beseparated by wavelength, rather than focal length, or in addition tofocal length. If so, at block 945, the captured images are separated bywavelength. This enables capturing data such as blood flow, and otherimage data not in the visible spectrum. The process then continues toblock 950.

At block 950, the process determines whether the data is designed forpolarity analysis. If so, at block 955, differential intensity analysisis used to generate this data, at block 955. This may be used to discernpolarity of things, based on the energy distribution from a lightsource. The differential intensity of two polarities may be used todetermine the elliptical polarities. The process then continues to block960.

At block 960, the process determines whether the data is designed forlight field reconstruction. Light field reconstruction captures all therays of light, so that each pixel captures not only the total lightlevel, but the directionality of the light. By capturing a plurality ofimages, at different focal distances, the directionality of each lightray may be used to reconstruct the light field, at block 965. Bycapturing the light field, the system can later adjust the focusarea(s). The process then continues to block 970.

At block 970, the process determines whether the data is for a 3D scan.A 3D scan reconstructs a three-dimensional image. By capturing aplurality of images at a plurality of focal distances, the system may beused to construct a 3D image. This is done at block 975.

In one embodiment, the system may also be used for 360 degree sphericalcameras. By capturing a sequence of images with different orientations,with uniform focal changes at the plurality of orientations, the systemcan capture data at a plurality of discrete focal steps, that can becalibrated for. This may enable easier stitching of images to create the360 degree image. Of course, this technique, like the others described,may be applied to video or image sequences as well as still images.

The process then ends at block 990.

FIG. 10 is a block diagram of one embodiment of a computer system thatmay be used with the present invention. The computer system may beutilized in setting the focal distances for image capture, forreconstruction of the image based on the captured image data, and forpost-processing the data to enable some or all of the features discussedabove. In one embodiment, the reconstructed image data may be thendisplayed, using the computer system. The display may be a near-eyedisplay or another type of display.

The data processing system illustrated in FIG. 10 includes a bus orother internal communication means 1040 for communicating information,and a processing unit 1010 coupled to the bus 1040 for processinginformation. The processing unit 1010 may be a central processing unit(CPU), a digital signal processor (DSP), or another type of processingunit 1010.

The system further includes, in one embodiment, a random access memory(RAM) or other volatile storage device 1020 (referred to as memory),coupled to bus 1040 for storing information and instructions to beexecuted by processor 1010. Main memory 1020 may also be used forstoring temporary variables or other intermediate information duringexecution of instructions by processing unit 1010.

The system also comprises in one embodiment a read only memory (ROM)1050 and/or static storage device 1050 coupled to bus 1040 for storingstatic information and instructions for processor 1010. In oneembodiment, the system also includes a data storage device 1030 such asa magnetic disk or optical disk and its corresponding disk drive, orFlash memory or other storage which is capable of storing data when nopower is supplied to the system. Data storage device 1030 in oneembodiment is coupled to bus 1040 for storing information andinstructions.

The system may further be coupled to an output device 1070, such as acathode ray tube (CRT) or a liquid crystal display (LCD) coupled to bus1040 through bus 1060 for outputting information. The output device 1070may be a visual output device, an audio output device, and/or tactileoutput device (e.g. vibrations, etc.).

An input device 1075 may be coupled to the bus 1060. The input device1075 may be an alphanumeric input device, such as a keyboard includingalphanumeric and other keys, for enabling a user to communicateinformation and command selections to processing unit 1010. Anadditional user input device 1080 may further be included. One such userinput device 1080 is a cursor control device, such as a mouse, atrackball, stylus, cursor direction keys, or touch screen, may becoupled to bus 1040 through bus 1060 for communicating directioninformation and command selections to processing unit 1010, and forcontrolling movement on display device 1070.

Another device, which may optionally be coupled to computer system 1000,is a network device 1085 for accessing other nodes of a distributedsystem via a network. The communication device 1085 may include any of anumber of commercially available networking peripheral devices such asthose used for coupling to an Ethernet, token ring, Internet, or widearea network, personal area network, wireless network or other method ofaccessing other devices. The communication device 1085 may further be anull-modem connection, or any other mechanism that provides connectivitybetween the computer system 1000 and the outside world.

Note that any or all of the components of this system illustrated inFIG. 10 and associated hardware may be used in various embodiments ofthe present invention.

It will be appreciated by those of ordinary skill in the art that theparticular machine that embodies the present invention may be configuredin various ways according to the particular implementation. The controllogic or software implementing the present invention can be stored inmain memory 1020, mass storage device 1030, or other storage mediumlocally or remotely accessible to processor 1010.

It will be apparent to those of ordinary skill in the art that thesystem, method, and process described herein can be implemented assoftware stored in main memory 1020 or read only memory 1050 andexecuted by processor 1010. This control logic or software may also beresident on an article of manufacture comprising a computer readablemedium having computer readable program code embodied therein and beingreadable by the mass storage device 1030 and for causing the processor1010 to operate in accordance with the methods and teachings herein.

The present invention may also be embodied in a handheld or portabledevice containing a subset of the computer hardware components describedabove. For example, the handheld device may be configured to containonly the bus 1040, the processor 1010, and memory 1050 and/or 1020.

The handheld device may be configured to include a set of buttons orinput signaling components with which a user may select from a set ofavailable options. These could be considered input device #1 1075 orinput device #2 1080. The handheld device may also be configured toinclude an output device 1070 such as a liquid crystal display (LCD) ordisplay element matrix for displaying information to a user of thehandheld device. Conventional methods may be used to implement such ahandheld device. The implementation of the present invention for such adevice would be apparent to one of ordinary skill in the art given thedisclosure of the present invention as provided herein.

The present invention may also be embodied in a special purposeappliance including a subset of the computer hardware componentsdescribed above, such as a kiosk or a vehicle. For example, theappliance may include a processing unit 1010, a data storage device1030, a bus 1040, and memory 1020, and no input/output mechanisms, oronly rudimentary communications mechanisms, such as a small touch-screenthat permits the user to communicate in a basic manner with the device.In general, the more special-purpose the device is, the fewer of theelements need be present for the device to function. In some devices,communications with the user may be through a touch-based screen, orsimilar mechanism. In one embodiment, the device may not provide anydirect input/output signals, but may be configured and accessed througha website or other network-based connection through network device 1085.

It will be appreciated by those of ordinary skill in the art that anyconfiguration of the particular machine implemented as the computersystem may be used according to the particular implementation. Thecontrol logic or software implementing the present invention can bestored on any machine-readable medium locally or remotely accessible toprocessor 1010. A machine-readable medium includes any mechanism forstoring information in a form readable by a machine (e.g. a computer).For example, a machine readable medium includes read-only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, or other storage media which may be usedfor temporary or permanent data storage. In one embodiment, the controllogic may be implemented as transmittable data, such as electrical,optical, acoustical or other forms of propagated signals (e.g. carrierwaves, infrared signals, digital signals, etc.).

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will, however,be evident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense.

We claim:
 1. An image capture system comprising: a lens directed at ascene to be captured; a modulation stack comprising one or more digitallight path length modulators, a digital light path length modulatorcomprising: a polarization modulator to receive light and to selectivelymodulate a polarization of the light; and an optical path lengthextender (OPLE) to direct the light having a first polarization througha first light path through the OPLE, and to direct the light having asecond polarization through a second light path through the OPLE, thefirst and second light paths having different light path lengths;wherein the modulation stack adjusts the path length of the light byaltering the polarization of the light between the one or more digitallight path length modulators to create a plurality of light pathlengths; and an image sensor to capture a plurality of image portions ata plurality of focal distances, from the data captured by the modulationstack, wherein a number of focal distances depends on the number of theone or more digital light path length modulators.
 2. The image capturesystem of claim 1, wherein a farthest virtual object distance isinfinity, and the focal distances move in discrete steps.
 3. The imagecapture system of claim 1, further comprising: a digital correctionsystem to separate simultaneously captured image portions with differentfocal distances.
 4. The image capture system of claim 1, wherein thepolarization modulator alters the path length on a time sequentialbasis, enabling the image sensor to capture a time sequential series ofimages at a plurality of focal distances.
 5. The image capture system ofclaim 1, further comprising: an auxiliary control system to enableselection of the plurality of focal distances for the image capture. 6.The image capture system of claim 1, further comprising: a displaysystem to display the plurality of image portions at the plurality offocal distances.
 7. The image capture system of claim 1, furthercomprising: a last OPLE in the modulation stack, the last OPLE includinga plurality of stripes, to enable simultaneous capture of image elementsat two focal distances with the image capture device.
 8. The imagecapture system of claim 1, further comprising: a camera body includingthe image sensor and a lens; and the modulation stack implemented in acamera lens assembly, enabling retrofitting of an existing camera formulti-focal image capture.
 9. The image capture system of claim 1,further comprising: a processor to process captured data from the imagesensor, the processor to enable one or more of: reconstructing thecaptured data based on one of polarization or wavelength, performwavelength-based analysis, reconstructing a light field based on two ormore images at different focal distances, generating a 3D scan based ona plurality of images at a plurality of focal distances, and creating amulti-focal 360 degree photo.
 10. The image capture system of claim 1,wherein the image captured is video, and the system continuouslycaptures video frames at a plurality of focal distances.
 11. The imagecapture system of claim 10, wherein the plurality of focal distances areadjustable during the image capture.
 12. An image capture systemcomprising: a modulation stack comprising one or more digital light pathlength modulators, a digital light path length modulator comprising: apolarization modulator to receive light and to selectively modulate apolarization of the light, wherein the polarization modulator alters thepath length on a time sequential basis, enabling the image capturedevice to capture a time sequential series of images at a plurality offocal distances; and an optical path length extender (OPLE) to directthe light having a first polarization through a first light path throughthe OPLE, and to direct the light having a second polarization through asecond light path through the OPLE, the first and second light pathshaving different light path lengths; and an image capture device tocapture a plurality of image portions at a plurality of focal distances.13. The image capture system of claim 12, further comprising: anauxiliary control system to enable selection of a focal distance for theimage capture.
 14. The image capture system of claim 13, wherein theauxiliary control system includes content-based focal distance selectionto automatically identify one or more focal distances for capture. 15.The image capture system of claim 12, comprising: a last OPLE in themodulation stack, the last OPLE including a plurality of stripes, toenable simultaneous capture of image elements at two focal distanceswith the image capture device.
 16. The image capture system of claim 12,wherein the modulation stack is implemented in a camera lens assembly,enabling retrofitting of an existing camera for multi-focal imagecapture.
 17. An image capture system comprising: an image capturesub-system including: a lens directed at a scene to be captured; amodulation stack comprising one or more digital light path lengthmodulators, a digital light path length modulator comprising: apolarization modulator to receive light and to selectively modulate apolarization of the light; and an optical path length extender (OPLE) todirect the light having a first polarization through a first light paththrough the OPLE, and to direct the light having a second polarizationthrough a second light path through the OPLE, the first and second lightpaths having different light path lengths; and an image sensor tocapture a plurality of image portions at a plurality of focal distances,from the data captured by the modulation stack, wherein a number offocal distances depends on the number of the one or more digital lightpath length modulators; an auxiliary control system to select theplurality of focal distances; a processor to process the captured data,to generate one or more of: a plurality of images at a plurality offocal distances, or a multi-focal image; a camera including the imagecapture subassembly; and a computer system including the processor,wherein the camera and the computer system are coupled using an I/Osystem.
 18. An optical path length extender (OPLE) comprising: aplurality of polarization sensitive reflective elements parallel to eachother and at an angle of between 20 degrees and 70 degrees to a path oflight entering the OPLE, to direct the light having a first polarizationthrough a first light path through the OPLE, and to direct the lighthaving a second polarization through a second light path through theOPLE, the first and second light paths having different light pathlengths; a plurality of light blocking stripes on an entry surface ofthe OPLE, to block light; wherein an output of the OPLE comprisesalternating stripes of image data, at a first and a second focaldistance; an image capture apparatus to capture image data at two focaldistances; and an image display apparatus to display data at two focaldistances.
 19. An image capture system comprising: a lens directed at ascene to be captured; an optical path length extender (OPLE) to directthe light having a first polarization through a first light path throughthe OPLE, and to direct the light having a second polarization through asecond light path through the OPLE, the first and second light pathshaving different light path lengths; a polarizer; a digital light pathlength modulator comprising: a polarization modulator to receive lightand to selectively modulate a polarization of the light; and the opticalpath length extender (OPLE) to receive the light from the polarizationmodulator; and an image sensor to capture a plurality of image portionsat a plurality of focal distances, from the data captured by themodulation stack.
 20. The image capture system of claim 19, wherein theOPLE further comprises: a plurality of polarization sensitive reflectiveelements parallel to each other and at an angle of between 20 degreesand 70 degrees to a path of light entering the OPLE; a plurality oflight blocking stripes on an entry surface of the OPLE, to block light;wherein an output of the OPLE comprising alternating stripes of imagedata, at a first and a second focal distance.